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DETAILED ACTION 

1 . This Office Action is responsive to the amendment received 16 December 2008. 

2. Claims 1-6 and 9 are cancelled in the current amendment, claims 7 and 8 were previously 
cancelled and new claims 10-16 are entered into the record and are currently pending for 
examination. 

Specification 

3. The abstract of the disclosure is objected to because the version of the abstract received 
16 December 2008 significantly exceeds the 150 word limit imposed upon an abstract. 
Correction is required. See MPEP § 608.01(b). 

4. The applicants' amendments have necessitated the new ground(s) of rejection which 
follow. 

Claim Rejections - 35 USC §103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3 . Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 
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5. Claims 1-9 (Cancelled). 

6. Claims 10 and 13-16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Jaszlics et al. (U. S. Patent 6,166,744, already of record, hereafter '744) and further in view of 
Meisner et al. (U. S. Patent 6,625,299 Bl, hereafter '299). 

7. In regard to claim 10 (New), Jaszlics teaches an image processing apparatus for 
composting an image of a virtual object and a physical space image to generate a mixed reality 
image ('744; fig. 1 & 2) but does not teach and causing an HMD to display the mixed reality 
image, and continues teaching comprising: a database ('744; fig. 22) which holds data used for 
generating the image of the virtual object; an image capturing unit ('744; fig. 2, element 100, 
camera) but does not teach which is attached to the HMD and continues to teach the camera 
captures the physical space image ('744; fig. 2; col. 8, In. 53-61); a first measurement unit which 
measures a position and orientation of the HMD ("744; head position; col. 10, In. 48-60); an 
object manipulation unit which is used by a user in order to operate a position and orientation of 
the virtual object ('744; joystick; col. 10, In. 35-38; col. 1 1 , In. 47-56); a second measurement 
unit which measures a position and orientation of said object manipulation unit ('744; col. 1 1, In. 
47-56); an operation panel which is positioned in a physical space ('744; fig. 16, Display A), 
displays an operation panel image ('744; fig. 21) used for editing the virtual object, and is 
capable of receiving a user instruction of editing the virtual object ('744; an operation panel 
image generation unit which generates the operation panel image by using the data held in said 
database ('744; fig. 20 & 22; virtual simulation computer); a rendering unit which updates the 
data held in said database according to the user instruction received via said operation panel and 
the measurement result of said second measurement unit, and renders, by using the updated data, 
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the image of the virtual object according to the measurement results of said first and second 
measurement units ('744; fig. 20 & 22; virtual simulation computer); a composition unit which 
composites the image of the virtual object rendered by said rendering unit and the physical space 
image captured by said image capturing unit to generate the mixed reality image ('744; fig. 20 & 
22; virtual simulation computer); but does not teach an HMD which displays the mixed reality 
image generated by said composition unit. Meisner, working in the same field of endeavor, 
however, teaches a HMD to display the mixed reality image ('299; fig. 15; col. 8, In. 1-10) and a 
video camera attached to the HMD for physical scene capture ('299; fig. 15; col. 8, In. 1-10) for 
the benefit of allowing the user hands free involvement in the VR simulation scenarios. It would 
have been obvious to one of ordinary skill in the art at the time of the invention to have 
combined the HMD teaching of Meisner with the rich VR apparatus teaching of Jaszlics for the 
benefit of allowing the user hands free involvement in the VR simulation scenarios. 

8. Regarding claim 13 (New), Jaszlics and Meisner teach the apparatus according to claim 
10 and Jaszlics further teaches wherein said operation panel includes a display device ('744; fig. 
16, Display A) and an operation device ('744; joystick; col. 10, In. 35-38; col. 1 1, In. 47-56), 
wherein the display device displays the operation panel image, and wherein the operation device 
is used for inputting the user instruction. 

9. In regard to claim 14 (New), Jaszlics and Meisner teach the apparatus according to claim 
10 and Meisner further teaches wherein the HMD can present the mixed reality image to two 
eyes of a user who wears the HMD ('299; fig. 15). 

10. Regarding claim 15 (New) Jaszlics teaches an image processing method of composting 
an image of a virtual object and a physical space image to generate a mixed reality image ('744; 
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fig. 1 & 2) but does not teach causing an HMD to display the mixed reality image, and continues 
teaching comprising the steps of: holding data used for generating the image of the virtual object 
in a database ('744; fig. 22); providing an image capturing unit ('744; fig. 2, element 100, 
camera), but does not teach which is attached to the HMD, and continues to teach the camera 
capturing the physical space image ('744; fig. 2; col. 8, In. 53-61); measuring a position and 
orientation of the HMD with a first measurement unit ("744; head position; col. 10, In. 48-60); 
operating an object manipulation unit, by a user, in order to position and orient the virtual object 
('744; joystick; col. 10, In. 35-38; col. 1 1, In. 47-56); measuring a position and orientation of the 
object manipulation unit with a second measurement unit ('744; col. 1 1, In. 47-56); positioning 
an operation panel in a physical space to display an operation panel image used for editing the 
virtual object ('744; fig. 16, Display A), the operation panel being capable of receiving a user 
instruction of editing the virtual object ('744; fig. 21); generating the operation panel image by 
using the data held in the database with an operation panel image generation unit; updating the 
data held in the database according to the user instruction received via the operation panel and 
the measurement result of the second measurement unit ('744; fig. 20 & 22; virtual simulation 
computer), and rendering, by using the updated data, the image of the virtual object according to 
the measurement results of the first and second measurement units ('744; fig. 20 & 22; virtual 
simulation computer); compositing the rendered image of the virtual object and the captured 
physical space image to generate the mixed reality image ('744; fig. 20 & 22; virtual simulation 
computer); but does not teach displaying the generated mixed reality image on an HMD. 
Meisner, working in the same field of endeavor, however, teaches a HMD to display the mixed 
reality image ('299; fig. 15; col. 8, In. 1-10) and a video camera attached to the HMD for 
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physical scene capture ('299; fig. 15; col. 8, In. 1-10) for the benefit of allowing the user hands 
free involvement in the VR simulation scenarios. It would have been obvious to one of ordinary 
skill in the art at the time of the invention to have combined the HMD teaching of Meisner with 
the rich VR apparatus teaching of Jaszlics for the benefit of allowing the user hands free 
involvement in the VR simulation scenarios. 

11. In regard to claim 16 (New), Jaszlics teaches a computer-readable storage medium 
encoded with a computer program for an image processing method of composting an image of a 
virtual object and a physical space image to generate a mixed reality image ('744; fig. 1 & 2) but 
does not teach causing an HMD to display the mixed reality image, and continues teaching 
comprising the steps of: holding data used for generating the image of the virtual object in a 
database ('744; fig. 22); providing an image capturing unit ('744; fig. 2, element 100, camera), 
but does not teach which is attached to the HMD, and continues to teaching to capture the 
physical space image ('744; fig. 2; col. 8, In. 53-61); measuring a position and orientation of the 
HMD with a first measurement unit ("744; head position; col. 10, In. 48-60); operating an object 
manipulation unit, by a user, in order to position and orient the virtual object ('744; joystick; col. 
10, In. 35-38; col. 1 1, In. 47-56); measuring a position and orientation of the object manipulation 
unit with a second measurement unit ('744; col. 1 1, In. 47-56); positioning an operation panel in 
a physical space ('744; fig. 16, Display A) to display an operation panel image used for editing 
the virtual object ('744; fig. 21), the operation panel being capable of receiving a user instruction 
of editing the virtual object('744; fig. 21); generating the operation panel image by using the data 
held in the database with an operation panel image generation unit ('744; fig. 20 & 22; virtual 
simulation computer); updating the data held in the database according to the user instruction 
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received via the operation panel and the measurement result of the second measurement unit, and 
rendering, by using the updated data('744; fig. 20 & 22; virtual simulation computer), the image 
of the virtual object according to the measurement results of the first and second measurement 
units('744; fig. 20 & 22; virtual simulation computer); compositing the rendered image of the 
virtual object and the captured physical space image to generate the mixed reality image('744; 
fig. 20 & 22; virtual simulation computer); but does not teach displaying the generated mixed 
reality image on an HMD. Meisner, working in the same field of endeavor, however, teaches a 
HMD to display the mixed reality image ('299; fig. 15; col. 8, In. 1-10) and a video camera 
attached to the HMD for physical scene capture ('299; fig. 15; col. 8, In. 1-10) for the benefit of 
allowing the user hands free involvement in the VR simulation scenarios. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to have combined the HMD 
teaching of Meisner with the rich VR apparatus teaching of Jaszlics for the benefit of allowing 
the user hands free involvement in the VR simulation scenarios. 

12. Claims 11-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Jaszlics et 
al. (U. S. Patent 6,166,744, already of record, hereafter '744) as applied to claims 10 and 13-16 
above and in view of Meisner et al. (U. S. Patent 6,625,299 Bl, hereafter '299) as applied to 
claims 10 and 13-16 above and further in view of Lescinsky et al. ("Interactive Scene 
Manipulation in the Virtue3D System," ACM Weg3D'02, pp. 127-135, February 24-28, 2002, 
Tempe, Arizona, ACM 1-58113-468-1/02/0002, already of record, hereafter '468). 

13. Regarding claim 1 1 (New), Jaszlics and Meisner teach the apparatus according to claim 
10 but do not teach wherein the image of the virtual object is generated on the basis of 3D CAD 
data of the virtual object, and said operation panel displays an assembly tree based on the 3D 
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CAD data. Lescinsky, working in the same field of endeavor, however, teaches wherein the 
image of the virtual object is generated on the basis of 3D CAD data of the virtual object, and 
said operation panel displays an assembly tree based on the 3D CAD data ('468; fig. 4, pg. 133, 
§9). 

14. Regarding claim 12 (New), Lescinsky further teaches wherein a part, which is obtained 
by enlarging a designated part of the assembly tree included in the operation panel image, 
includes a component name contained in the assembly tree ('468; fig. 4, pg. 133, § 9). 

Response to Arguments 

15. Applicant's arguments with respect to claims 10-16 have been considered but are moot in 
view of the new ground(s) of rejection. 

Conclusion 

The following prior art, made of record, was not relied upon but is considered pertinent to 
applicant's disclosure: 

US 6972734 Bl Mixed reality apparatus and mixed reality presentation method 

US 20010038378 Al Portable game display and method for controlling same - An 
augmented-reality display method and system for the visually 
impaired. 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
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the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Edward Martello whose telephone number is (571) 270-1883. 
The examiner can normally be reached on M-F 7:30-5:00 EST. 

If attempts to reach the examiner by telephone arc unsuccessful, the examiner's 
supervisor, Xiao Wu can be reached on (571) 272-7761 . The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/EM/ 

Examiner, Art Unit 2628 
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Supervisory Patent Examiner, Art Unit 2628 



